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In @ slaae dewigned to guntaute oral language : 
proficiency tests, it is planned to administer a series of tests to 
100 native Mandarin Chinese-speaking subjects (foreign students and ae 
their spouses). The tests will measure communicative competence in ' 

‘speaking (ability to speak, exhibiting centrol of linguistic, 

‘ sociolinguistic, and pragmatic rules: and fluency) and communicative 
competence in reading (ability to react to these rules as manifested 
in written language, and to react fluently). Three different testing 
methods will be used, resulting in a multitrait-multinethod design: - 
interviews, translation, and self-rating. The results will verify 
‘hypotheses of competence, and the ita alle of the construct, oral 
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aS * secently, ¢onsiderable research has been devoted to ‘testing oral 


= language proficiency, and a number of differert oral testing procedures o 
. have emerged. (Clark, 1975, 1978, 1979; Jones, 1975,‘1979; Palmer and = 
Groot, 1979). Central to much of this research is the acceptance of * 


"face "validity" as a criterion for evaluating oral proficiency tests, 
and the reliance on concurrent validation procedures for relating "in- 
direct" to "direct" testing methods. - 
Both of these approaches to validity have been shown to be of 
dubious utility (Cronbach and Meehl, 1955; Stevenson, 1979). The pro- ° ; * 
blems inherent in criterion-referenced validation include not only the 
difficulty of establishing an adequate criterion measure, but also the 
potentially serpentine process of successive approximation. ,While: 

- circular validation procedures are generally precluded by conscientious, 
‘test developers, the problem of valid criterion measures remains. The aad 
proposed solution to this problem in the area of language proficiency 

¢ testing has been the appeal to the "face validity" of so-called "direct" 
measures (Clark; 1975, 1979; Jones, 1975). The notion of "face validity" 
in the case of language proficiency is intuitively appealing. Obviously 
the most direct sample of speaking proficiency, for example, is for some- 
one to speak. That is, the most direct sample of a given behavior is the. 
behavior itself. This becomes: less obvious, however, when we consider 
another mental trait,-intelligence. No one, I believe, would claim that 
digit-symbol substitution, which is one test in the Wechsler Adult In- 

‘ telligence Scale, can be equated’ with intelligence. That is, we do not 
posit the identity of a trait with its behavioral manifestation. The 
problem of "face validity," then, as it: has been advocated in language 
proficiency testing, ts that it confuses the outward manifestation of a 
trait with the trait itself. Once this is recognized, the distinction 
between "direct" and "indirect" measures becomes irrelevant, since all 
tests sample manifestation&\of traits, and not the traits themselves. 
With this recognition that a}l se are indirect measures of traits, the 
notion of "face validity" bedomes "the mere appearance of validity,’ [and] 
is not an acceptable basis for interpretive inferences from tést scores." , 

(APA, 1974, p. 26).° Therefore, the claim that a given test is.a valid 

measure of a given trait cannot be accepted on the basis of afi appeal to : = 
"face validity," but must be supported by a much more zigonods inference 
SESISEURE, that of construct validation, 
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In construct validation, or the process of investigating what psy- 
‘chological constructs are measured by a given test,‘a test is validated - 
not against another test, but against a theory. To investigate construct’ 
validity, one develops a construct (a theory)» which becomes a provisional 
explanation of test results until the theory is falsified by the results 
of testing hypotheses derived from it. Thus the test becomes , in essence, 

| the aperational definition of the construct. °- 


ec 


j design requires that at least two distinct traits be measured by at least 


Ye model of construct validation to be followed in this study -is ~ 
the ff titrait-multimethod matrix (Campbell and Fiske, 1959). This 
degign recognizes that any test score is a function of both the trait it ¢ 
Aitends to measure and of the method by which it is,measured. In order - ‘ 
fto distinguish trait variance from method variance in test scores, the ; 


two separate methods. High correlations between scores on different i 
measures of the same trait would demonstrate convergent validity. High 
correlations between similar methods -of measuring different traits,, 
however, would invalidate the test, and so we interpret low correlations 
between such measures as evidence of discriminant validity. In the multi- 
trait-multimethod matrix, method variance can thus be delineated from 
trait variance, so that both convergent and discriminant validity’can be 
examined. ' : \w ; 
The basic research design of this study is a 2x3 multitrait- 
multimethod matrix, with the following traits and methods given below. 
Trait definitions, are derived from the geheral framework developed by,_ 


e ‘ »« Canale and Swain (1979). e : 


Traits ; % ; a 


: ¢ 


A. Communicative competence in speaking consists of: 


1. The ability to produce spoken language exhibiting control of the 
linguistic rules employed by spealers of a given dialect or set 
of dialects. ‘Control consists of: breadth (the range ‘of SEEUGEUE es 4 
attempted) and accuracy (the degree to which structures are ra 
ced correctly). The areas of veers control are ayny 


du 
__ Phonology, and lexicon. ~ one 
. if: r 
‘ oe The. ability to produce spoken language exhibiting control of the 
‘ sociolinguistic rules employed by speakers of a given dialect or . yf 
set of dialects. ‘Sociolinguistic rules consist 6f the conventions 
for producing speech in a register appropriate’ to specific speech 
_ situations. Control consists of breadth (tke range of speech 
situations in which the speaker is sensitive to different prevail- 
ing standards) and accuracy (the degree’to which the language 
produced conforms to prevailing standards). 


3. The ability to produce spoken language which exhibits control of 
the pragmatic rules employed by speakers of a given dialect or 
set of dialects for comm cating the typ f messages required’ 
, by these speakers. Pra tic rules /are conventions for velnting * : 
» the form of an utterap e to its intended meaning. Control consists 
of breadth (the range and complexity of messages communicated) and 
accuracy (the degt to which the language produced coraneenee 
correctly the. det ite of the a 


ov 


4. The ability to aisles spoken language fluently. Fluency consists 
- of quickness of response to perceived needs to speak and the rate 
of speech (the degree to which the tempo of the speech conforms in 
overall speed and, consistency of speed to norms for speakers of a 
given dialect or set of dialects). , 
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B. Communicative competence in reading consists of: 4 
‘ : ' : a Pi 
’ ‘ 1. The ability to react to the linguistic rules manifested in | 
_ written language. Ability consists of breadth (the range of 
structures reacted to) and accuracy (the degree to whith re- 
r actions are correct). Areas of linguistic.control are graphology, *' 
syntax and lexicon. | ft ood he 
‘2. The ability to react to the sociolinguistic rules employed in a 
given written dialect or set of dialects. Sociolinguistic rules 
consist of conventions appropriate to’ particular aims and modes 
. of written discourse. Control consists of breadth (the range of 
aims.and modes in which the spéaker is sensitive to prevailing 


a standards) and accuracy (the” gree to which reactions conform to 
i prevailing standards). ; ine : 


3. The ability to react to the pragmatic rules employed in a given 
written dialett or set of dialects to communicate, types of 
a messages appropriate to that. dialect or set of dialects. Prag- 
matic rules are conventions for relating the form of a text to 
its intended message. The ability to react consists of breadth 
Athe range of messages) and accuracy (the degree to which the 
“reactions conform to prevailing standards). 


a 


: i 
: ‘ ’ 4. The ability to react to written’ language fluently. Fluency con- 
. nF sists of the rate of response to written material (the degree to 
- which responses conform to norms for readers of a-given dialect 
e or set of dialects). . 
4 “/ ¥ 
Methods * ay 


A. The interview method consists of a face-to-face language use situation 
requiring subjects to interact with one or more interlocutors,, ex- 
changing §nformation, but requiring no direct translation from the. 

ao subjects’ native language ‘to the target language or vice-versa. 

B. The translation method consists of a language use situation requiring 
the subjects to*translate directly from their native language to the 

ae target language and/or vice-versa. The situation is not face-to-face 
eand there is no interaction with an interlocutor. 


C. The self-evalugtion method jconsists of subjects’ self=tatings in their 
native language of their ability in the specified traits in the target 
language. There is no use of the target language, no direct transla- 
tion, and no interaction. This design can be schematized as in Figure 


1. below. : 
oN Method 
(Self-evaluation 


| fini) |. Giemstavion| 

: res Interview) Translation 

Trait A : os . ; bag ° 
— 7 


Figure 1 


 ‘Trait-method units in the multitrait-multimethod design of the study 
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Trait-method unit A; will consist of a standard FSI-type oral inter- 
view. This highly Structured interView consists of several distinct 
parts and includes well-defined procedures for checking the subjects’ 
levels of proficiency and for probing to determine the upper bounds of ’ 


, these levels. /(Wilds, 1975; Lowe, 1976). Research indigates that this 


testing procedure has high reliability (Adams, 1978; Clifford, 1978; 
Mullen, 1978), predictive validity (Clark, 1975; Jones, 1978), and high 


’ concurrent validity (Shohamy, 1979; Hendricks et al., -1979). Interviews 


will be conducted with two interlocutors. Simultaneous ratings, both , 
individual and conference, will be given. In se interviews will 
be tape-recorded, arranged in random order, and ratéd at a later time. 
Trait-method unit A, wiIl consist of a series of aioe dialogues in 
the subjects" native language which they will listen to and then provide 
a direct’ oral‘translation into English’. These -didlogs will vary in ways 
consistent with the types of controt specified in the definition of 


speaking. The'translations will be recorded .on tape, arranged in random | 


order, and rated, hci a scale consistent with the FSI rating scale. 


Trait-method unit re will consist of the waubanete! self-rating of 
their oral proficiency in English. Both Lickert and semantic-differen- 
tial scales will be used. a Se , 
-* Trait-method unit B, will consist of a set of. graded reading’ pass- 
‘ages in English, which the subjects will read. An interlocutor will then _ 
ask questions about the passage/in the subjects! native language, which 
they will. answer in their nati¥e language. ‘discussion will focus on 
the subjects’ comprehension the reading assage, and no direct trans- 
lation will be requlnrte interview will be recorded and rated at a 
later time. : / . 


ae 

Pn alee unit will consigt of a set of graded reading pass- 

ages in English which the subjects will read silently and then translate 
directly, line by line//.into their/native language. These translations 

"will be tape recorded/And rated af a later time. : 


Trait-method unfit B3 will consist of the subjects' self-rating of 
their reading profiéiency in English. Both §tckert and semantic- 
differential scalés will be used. | _ a. , i. 
/ $e 5 

Subjects fof this study will be limited to a homogeneous mother- 
tongue group of/non-native spéakers of English. The ‘intended sample of 


the interrelationships among the various trait-method units 
rix. ‘One method for doing this is to compute a matrix of 


thr¢e other analytic procedures have been propoged: analysis of vari- 
ye (Mellon and Crano, 1977), confirmatery factor analysis. (J8reskog, 
er 9% pales and Riuesel., 1975) is aaa criteria (Althauser, 


. 2 - 


\s 


Method 1 Method 2° 
Traits , 


‘Method 1 Ay 


B) 


Method 3 


Figure 2 
Idealized 2x3 multitrait-multimethod matrix: 


r = reliabilities (monotrait-monomethod correlations) - : 


c = " convergent validfties (monotrait-heteromethod correlation 


ds = discriminant correlations (heterotrait-monomethod) . 


- 


d. = discriminant correlations (heterotrait-heteromethod) 
nop Bate. ee 


In this matrix, we can identify a diagonal row of correlations between 


the same method and the same trait. These monotrait-monomethod correla- : 


tions (r) comprise the, reliabilities of the trait-method units involved. 
Two other diagonals parallel to the reliability diagonal consist of 
correlations between different methods of ‘measuring the same trait. 
These monotrait-heteromethod correlations (c) comprise convergent valid- 
ity coefficients. The two other sets of correlations are 1) those. be- 
tween different methods of measuring different traits’ (heterotrait- 


.heteromethod--d},) and 2) those between the same method for mennur ing 


different traits BERS ES G mnOMetiedo tay > 


The logic of inferences toffbe made from urceebenei data 
requires two assumptions: ‘First, we assume that random error variance 
approaches zero. This assumption necessitates high reliabilities for 
all tests. If this requirement is not met, subsequent inferences from 
the data are highly questionable. . The second assumption pertains to f 


Heberlein and Scott, 1971),° These, procedures, in addition to the ex- 
amination of correlations described by Campbell and Fiske, will be , 
Roe erens particularly ‘in the follow-up studies suggested below. 
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; 
fhon-random error variance, and is two-fold: - method and-trait factors | 
are uncorrelated, and methed variance is constant across traits. (Alwin, 
1974). These latter assumptions must be met if inferences regarding 
areceiarnane validity are to. be valid. 


The hypotheses and dneroaiebe ‘from multitrait-multimethod ‘data are 


ns © as follows: .° ' 
:. | 
le ‘© 0 73 
> 
a Monotredt-heteronethod correlations (c) should be sigitti<| 
mH » cantly higher than zero, and "sufficiently large to encourage 


- further examination of ualidity " (Campbell and Fiske, p. 33) 
High correlations between different methods for measuring the 
same trait are seen as evidence of convergent validity. W 
monotrait-heteromethod correlations (c) jindicate lack of con- 
vergence and preclude further examination of discriminant 
validity.. : = 
. ; 2. ¢>dh F : 
' Convergent validity coefficients (c) should be higher tha 
' .the correlation obtained between different methods for measur- 
ing different traits (dp). Low heterotrait-heteromethod ¢corre- 
lations (dh) are inearpreted as evidence for, discriminant 
validity. 


¢ 


“3, c > dn 


; Convergent validity coefficients (c) should also be higher 
s than the correlations obtained between different traits | 
measured by the same:method (dm). Intuitively, high hetero- 
trait-monomethod correlations (dm) would indicate dominance 
- ' of method, and hence invalidate the test. Low heterotrait- 
, r jeonawethod correlations dre interpreted as additional evidence , 
' of discriminant validity. 


4, Similar patterns of traft interrelationships in all hetero- 


4 trait groups. For ewample, if the rank-order of correlations 
in one grouping is c > dh > dm, we would expect to find the 
=. ; ; same order in other such groupings. 
a 


Within this framework, the hypotheses of this project pertain to 
pr the following general questions regarding language proficiency: 


a : 1. Is there evidence that the trait "communicative competence 


_ in'speaking" is distinct from the trait "communicative com- 
y, ees in reading"? 


to both? 


» 
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The hypothesis of distinct traits (speaking and reading) will be 
supported if the data show evidence of both convergent and discriminant 
validity. Specifically, if we find 1) high correlations among the 
three methods on the same traits and 2) lower correlations among differ- 


, ent methods of measuring different traits and among the same methods for 


measuring different traits, we will have evidence to support the hypoth- -: 
esis of distinct skills. In this case, the ‘analysis of additional 
ratings of pronunciation, grammar, fluency and vocabulary will be, in- 
cluded in the matrix as traits, and analyzed to determine their pei are 
ness. If only convergent but not discriminant validity is evidenced,’ «' ~ 
then the hypothesis of distinct skills will not be supporteg. In aoe 
case, the analysis of. additional ratings may lead to hypotheses regarding 
factors common to both speaking and reading. Such analyses will provide 


more precise definitions of the traits examined. These definitions:in ‘. 


turn will form the basis for hypotheses of subsequent research into “the 
components ‘of communicative competence. 
‘( 


In this paper, we have argued that criteria currently folkowed for 
evaluating the validity of language-proficiency tests are inadequate. 
We have presented a specific model and set of procedures for investigat- 


ing both convergent and discriminant validity. In the study, presently 


being conductéd, a widely accepted and used procedure for testing oral 
language proficiency, the FSI oral interview, will be examined-for con- 
struct validity. A definition of oral proficiency based on a model of 
communicative competence is proposed as a framework for stating hypoth- 
eses. The results of this study will bear upon the unitary factor 
hypothesis of language proficiency. Further research will investigate 
the comporents of communicative competence, both in separate skill areas 
and in general. et aes oe 
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